The equivalent channel matrix dimension of Generalized Frequency Division Multiplexing with Multiple Input Multiple Output (MIMO-GFDM) system is very large, and the traditional Multiple Input Multiple Output (MIMO) detection algorithm has high complexity and poor performance. Aiming at those problems, Reactive Tabu Search (RTS) detection algorithm in massive MIMO systems was applied to MIMO-GFDM system, and the high complexity problem of the initial value in RTS algorithm was also solved. Firstly, by using the positive definite symmetry of the matrix used in Minimum Mean Squared Error (MMSE) detection algorithm, Cholesky decomposition was applied to the matrix, and Sherman-Morrison formula was combined to iteratively calculate the initial value, reducing high complexity of the initial value inversion. Then, with the result of the improved MMSE detection as the initial value of RTS algorithm, the optimum solution was searched globally from the initial value. Finally, the iteration numbers and Bit Error Rate (BER) performance were researched through simulations. Theoretical analysis and simulation results show that, in MIMO-GFDM, the improved RTS signal detection algorithm has much lower BER than traditional signal detection algorithms. In 4 Quadrature Amplitude Modulation (4QAM), the RTS algorithm has approximately 6 dB lower signal-to-noise performance gain than MMSE detection (when BER is 10 ^-3). In 16QAM, the RTS algorithm has approximately 4 dB lower signal-to-noise performance gain than MMSE detection (when BER is 10 ^-2). Compared with the traditional RTS algorithm, the proposed algorithm has lower complexity without affecting the BER performance.

In Spatial Modulation (SM) system, the Maximum Likelihood (ML) detection algorithm with the best performance has high complexity, while the complexity can be reduced by M-algorithm based on QR decomposition (QRD-M) of channel matrix. However, when the traditional QRD-M algorithm is used, fixed M nodes were chosen at each layer, which leads to additional computation. Therefore, for the problem of the traditional QRD-M algorithm, a Low-Complexity QR-Decomposition M-algorithm with dynamic value of M (LC-QRD-dM) was proposed. In LC-QRD-dM, by comparing the designed threshold with the cumulative branch metrics, the number of reserved nodes that does not exceed M was adaptively selected at each layer, thus reducing the computational complexity with the cost of a small amount of performance. Then, concerning the high bit error rate of LC-QRD-dM with deep channel fading, QR-Decomposition M-algorithm with dynamic value of M based on Channel State (CS-QRD-dM) was further proposed. Based on the principle of LC-QRD-dM, the number of reserved nodes that do not less than M was selected by the threshold value at each layer when the Signal-to-Noise Ratio (SNR) is not high; and the number of reserved nodes that do not exceed M was selected by the threshold value at each layer when the SNR is high. Theoretic analysis and simulation results show that, compared with the traditional QRD-M algorithm, CS-QRD-dM achieves about 1.3 dB SNR advantage (when the bit error rate is 10 ^-2) at low SNR, which can significantly improve the detection performance at the cost of small complexity increase; and its detection performance and complexity are the same as LC-QRD-dM at high SNR.